Discussions
Solve problems, and share tips and tricks with other JMP users.
Accepted Solutions
Hi @mjz5448,
I might add a complementary answer to elucidate some of the informations given, based on the previous responses I have already written on similar topics:
Predictor Screening is a method based on the Random Forest algorithm that help you identify important variables (identified by the calculation of features importance) for your response(s) in your historical data. Depending on the representativeness of your data, coverage of the experimental space, missing values, presence of outliers, interactions, correlations or multicollinearity between your factors, this platform may have some shortcuts, many unknowns, and shouldn't be trust blindly (as stated by @dlehman1).
- The analysis and insights you can have with this analysis is only limited to the historical experimental space (which might be too limited and/or not representative for a good understanding of the system). You can see this as Production experimental space (historical data) vs. R&D (DoE) : in production, only small variations of the factors is allowed (and some are fixed, so you won't be able to determine their importances) to create robust and quality products, whereas in R&D the objective is different : explore an experimental space and optimize the factors of a system, by using larger ranges for the factors considered.
- The Predictor screening platform doesn't give you a model (unlike the Bootstrap Forest available in JMP Pro), only feature/factor importances, so you don't have access to individual trees and branches, which could help identify interactions between factors, and the threshold levels values of factors used for the splits.
- Since you don't know the quality of the model behind (through R², RMSE or other relevant metrics according to your objectives), you don't know how much variability is explained through it (or how accurate the model is) and if the calculations of feature importance is relevant/adequate for your topic.
- Since all factors are ranked, the threshold/limit between important and non-important factors is not trivial. To help you with, you can add a random factor (with random values). An example of this technique is shown here with the Diamonds Dataset from JMP with Price as response, adding a random factor and omitting Carat Weight as possible input (since it is a very important predictor, so I wanted to show you an example where it's hard to sort out the other predictors) :
- This platform doesn't give you indications about possible correlations or multicollinearity between factors. You can use the Multivariate platform to explore correlations between inputs, outputs, and input-outputs. You can also create dummy regression models to calculate VIF scores between your factors, in order to spot multicollinearity.
- High cardinality categorical factors may also bias the feature importance calculations. The more cardinality (number of classes), the higher the bias, as the data will form a high number of small groups/classes that the model will try to learn. So it will fail to generalize or "understand the logic" and as a result this will increase the chance of overfitting, giving this feature a high importance.
An example here with the same dataset, same conditions but adding a random categorical factors with 11 levels (from A to K, named "Shuffle[Column10]). This categorical random feature with high cardinality is ranked higher than the random numerical variable seen before :
Concerning your questions about multicollinearity :
- The multicollinearity among variables is handled in Random Forest by the random feature subset selection at each node: at each node, to determine the next split, you randomly create a subset of your features and evaluate in this subset which feature (and at which threshold level) is able to separate best the data points based on their response values. So as every feature has equal chance to be selected, even correlated features can be selected and have a similar feature importance in the model. This process minimize the impact of correlation between columns, and is not present for Decision Tree (or Boosted Tree), where a single tree is created with all features tested at each node : only one of the correlated features might be selected for the split.
- Since Random Forest creates a big model based on many decision tree trained in parallel on bootstrap samples, each individual tree is trained on a slightly different dataset, which also help reducing the correlations between each trees (correlations between rows) and reduce model variance.
More infos on tree-based models :
And about the advantages of Random Forests : https://www.linkedin.com/posts/victorguiller_doe-machinelearning-randomforests-activity-712755779981...
Hope this complementary answer will help you,
Hi @mjz5448,
I might add a complementary answer to elucidate some of the informations given, based on the previous responses I have already written on similar topics:
Predictor Screening is a method based on the Random Forest algorithm that help you identify important variables (identified by the calculation of features importance) for your response(s) in your historical data. Depending on the representativeness of your data, coverage of the experimental space, missing values, presence of outliers, interactions, correlations or multicollinearity between your factors, this platform may have some shortcuts, many unknowns, and shouldn't be trust blindly (as stated by @dlehman1).
- The analysis and insights you can have with this analysis is only limited to the historical experimental space (which might be too limited and/or not representative for a good understanding of the system). You can see this as Production experimental space (historical data) vs. R&D (DoE) : in production, only small variations of the factors is allowed (and some are fixed, so you won't be able to determine their importances) to create robust and quality products, whereas in R&D the objective is different : explore an experimental space and optimize the factors of a system, by using larger ranges for the factors considered.
- The Predictor screening platform doesn't give you a model (unlike the Bootstrap Forest available in JMP Pro), only feature/factor importances, so you don't have access to individual trees and branches, which could help identify interactions between factors, and the threshold levels values of factors used for the splits.
- Since you don't know the quality of the model behind (through R², RMSE or other relevant metrics according to your objectives), you don't know how much variability is explained through it (or how accurate the model is) and if the calculations of feature importance is relevant/adequate for your topic.
- Since all factors are ranked, the threshold/limit between important and non-important factors is not trivial. To help you with, you can add a random factor (with random values). An example of this technique is shown here with the Diamonds Dataset from JMP with Price as response, adding a random factor and omitting Carat Weight as possible input (since it is a very important predictor, so I wanted to show you an example where it's hard to sort out the other predictors) :
- This platform doesn't give you indications about possible correlations or multicollinearity between factors. You can use the Multivariate platform to explore correlations between inputs, outputs, and input-outputs. You can also create dummy regression models to calculate VIF scores between your factors, in order to spot multicollinearity.
- High cardinality categorical factors may also bias the feature importance calculations. The more cardinality (number of classes), the higher the bias, as the data will form a high number of small groups/classes that the model will try to learn. So it will fail to generalize or "understand the logic" and as a result this will increase the chance of overfitting, giving this feature a high importance.
An example here with the same dataset, same conditions but adding a random categorical factors with 11 levels (from A to K, named "Shuffle[Column10]). This categorical random feature with high cardinality is ranked higher than the random numerical variable seen before :
Concerning your questions about multicollinearity :
- The multicollinearity among variables is handled in Random Forest by the random feature subset selection at each node: at each node, to determine the next split, you randomly create a subset of your features and evaluate in this subset which feature (and at which threshold level) is able to separate best the data points based on their response values. So as every feature has equal chance to be selected, even correlated features can be selected and have a similar feature importance in the model. This process minimize the impact of correlation between columns, and is not present for Decision Tree (or Boosted Tree), where a single tree is created with all features tested at each node : only one of the correlated features might be selected for the split.
- Since Random Forest creates a big model based on many decision tree trained in parallel on bootstrap samples, each individual tree is trained on a slightly different dataset, which also help reducing the correlations between each trees (correlations between rows) and reduce model variance.
More infos on tree-based models :
And about the advantages of Random Forests : https://www.linkedin.com/posts/victorguiller_doe-machinelearning-randomforests-activity-712755779981...
Hope this complementary answer will help you,