Hi @tonkatsu2020 ,     Something you might want to consider is whether or not your data table is "oriented" or arranged in the right way. Normally, you'd want to have a single column with data and then the rows would be each new instance of whatever it is you're looking at.     You might consider transposing your data. Then, it's just easy to find the Col Maximum() and Col Minimum(). However, this might not be what you're after. Can you share an anonymized version of your table? That might help in understanding how your data is structured and what you want to do with it.   DS     ... View more

Hi @Maureen ,     It sounds like you want to do something like a Profit Matrix or Cost Sensitive Learning. This would be done by doing a nominal logistic fit in the fit model platform.     Either that, or you would have to replicate your experiment several times to gather enough statistics for each method that you can determine p-values for the summary stats of sensitivity, specificity, etc.     For example if Run 1 of the experiment results in: Ref  = TP A = FP B = TP C = FN   But, if you repeat Run 1 again (after you've gone through all the other runs), maybe you get: Ref = TP A = FP B = FN C = TP   This would result in the numerical count being different for Run 1 for the different methods. As a result, if you then do a Fit Y by X, where the y-axis is the counts for each of the different runs and the x-axis is the different methods (ref, a, b, c), then each method (except maybe the reference method) should have a spread in the counts, which would result in slightly different summary statistics. The ANOVA would look at the means and standard deviations of those summary statistics to find if there are any significant differences between the methods. Since you'd be comparing four different methods, you'd want to do a Tukey-Kramer analysis of means rather than a Student's-t.   Another option would be to look into a Bland-Altman analysis. You'd have to recode the TP, TN, FP, FN to actual numerical values. See for example here. This analysis compares the difference of one method (to the reference) and compares it to the average of the reference to the measurement. In other words, it would look at DIF = Ref-A (on the y-axis) and AVG = (Ref+A)/2 (on the x-axis). You could do this analysis on the FP, TN, etc. or the summary statistics.   Depending on how many rows (experimental runs you have), I think you might be better off doing an analysis like in the first paragraph.   As mentioned before, if you can share your data table (you can always anonymize it (Tables > Anonymize)), it would be a lot easier to determine exactly how to help you.   Good luck!, DS ... View more

Hi @Maureen ,     I think I understand how your data is organized. If I do, and if I understand how you are trying to summarize each method's statistical evaluation, you're going to need to do some data table manipulation to get it in the right format. Here's what I'd do: Make a summary of your stacked data where you put the column containing the TP, FP, TN, FN in the "Group" entry and then the column that contains the information whether the row was reference, method A, method B, or method C as "Subgroup" (be sure to uncheck the "link to original data table" box). This creates a new data table with a column of the FN, TN, etc. characters, a column "N Rows" that says how many rows had that value and then several other columns with names like N(Method A), N(method B), N(reference), etc. These columns contain the counts of each of the TP, FP, TN, FN events. You'll need to then transpose this table. You'll want to put all the columns that are titled "N(name of something)" in the "Transpose Columns" selection and then for "Label" you'll want to use the column (maybe named "Data") containing the TP, TN, etc. values. This will generate a new data table that has a column called "Label", which will contain the column names from the previous table, i.e. "N(method A)" and so on. There will now be columns with the names "TP", "TN", "FP", "FN" and each row will contain the count of the number of times that particular event occurred for the specific method. Now, you will create your statistic summary columns as formulas. Specificity will be TP/(TP+FN), and so on for each of the summary statistics that you want to calculate. The last step would be to plot using the Fit Y by X platform. You'd use the "Label" column as the X Factor (this is the column containing values like "N(method A)", etc. and your summary statistic (sensitivity, specificity, etc.) as the Y Response. Unfortunately, since this is a summary statistic and not looking at each row's statistic you won't be able to do an ANOVA on the data. You will be able to see the overall picture for each method and how good each method is for the different statistics.   I hope I've understood the problem correctly. Again, it helps to be able to see the data structure to know exactly how to move from where you are to where you want to be.   Hope this helps!, DS ... View more

Hi @Maureen ,     There's a couple ways to go about this, and it depends on how your table's layout. If it's stacked, meaning you have a column labeled "Method" for example, and it has possible entries of "reference", "Alt A", "Alt B", etc., then you would create a statistic columns for each of the stats you're looking at, sensitivity, specificity and so on. Where each stat is calculated by the formulas you have. You would then do an ANOVA on the stacked column for sensitivity, etc. The ANOVA analysis should give you an idea if there is a significant difference in the sensitivity of A vs B vs C, etc. Having it stacked will really help to see not only how each of the three alternative methods compare to the reference, but how they all compare to each other.     If your table is arranged row-wise by the experimental run, then you'd want to create a new data tables where you stack by the stat columns in order to run the ANOVA.     It might be easier to address your specific issue if you're able to share the data table (anonymize it if need be) to better understand the data structure and where you're trying to go.     Out of curiosity, are your TP, FN, TN, FP values binary 0 or 1, or do they range between 0 and 1? Also, are you wanting to sum the stats (sensitivity, etc.) across all runs of the experiment to compare the aggregate values, or evaluate it on a run by run basis?   Hope this helps!, DS ... View more

Hi @34South,     XGBoost is really powerful and very stable. If you re-run it multiple times, you get the same answer, which isn't always the case for bootstrap forest or boosted tree.     I found this website to be VERY helpful on how to tune XGBoost and improve the models. And, since it's written for people running Python, you should be good to go.     Have you considered scripting your fit routines so that you can automate some of the process and can let run unsupervised? It might be quicker in the long run. I do that when searching the tuning parameter space for the other tree-based platforms in JMP -- often times running 30k's + of calculations. Depending on the platform, it can take a good 8-10 hrs, but I just let it run while I do other stuff.     You'll definitely want to monitor your R^2 for training and validation to make sure you're not overfitting the data. I'm trying to run some models now, but I'm finding that so far with the Neural Net and GenReg platforms, it's coming back with R^2 that are larger for the validation set than the training set, which is a big warning sign. Personally, I think it's because of the data set that I have and the highly non-normal distribution of the data.     One very nice thing about setting the test data set aside -- actually removing those data from the training data table so there's no way JMP might get access to them when running the fit. I have had the experience before that even though I hid and excluded certain rows, the data was still used during cross-validation, and so when comparing an actual vs predicted plot for the test data set it looked better than it actually was because some of the test data was being used in the training.   DS ... View more

Hi @34South,     Do you have JMP Pro or access to JMP Pro? I ask because the partition platform isn't the best (most stable) approach to generate prediction formulas. The Boosted Tree and Boostrap Forest platforms are much better. Or, if you have access to higher versions, you can also run the XGBoost platform, which is very stable.     But, on to your question about min size split. The min size splits controls complexity of the tree "stump". If the number of observations in a split is less than the value that you set, there is no split generated. It is not too relevant when you use boosting in your decision tree, but can be used to control it. You don't want it too small (i.e. 1 observation) and also don't want it too big (e.g. 1000 observations). A good typical staring point would be around 5 for a simple tree method. If you're using something like the bootstrap forest platform, you'll want to use N/2 for classification schemes like you're using -- where N is the number of observations. You might be able to tune your model by adjusting this parameter and then comparing it the different models on a test data set.     If you have enough data, I'd recommend splitting off a portion, maybe 20-25% and keeping that as a test data set that isn't used to train or validate your model. With the remaining data, you can generate a stratified validation column with training and validation data (say 80/20) to optimize the complexity of your model while also keeping it from overfitting. After generating several models, you can use the previous test data as a way to compare which model is actually better at predicting the outcomes correctly. Since you're looking for a binary decision model, you can even optimize it further by looking at cost sensitive learning.     I'm not an expert in how JMP calculates the log worth or Fischer's exact test, so hopefully someone from JMP can answer that, but I'm guessing that it has to do with the whole data set and calculating log worth versus when you save the split formula and do a contingency table.   Hope this helps -- or at least gives some things to think about.   Happy modeling!, DS ... View more

Hi @P_Bartell,     Thanks for the links, those are very interesting reads. The statapults are a bit pricey, but look very cool. From the manual, the design seems pretty simple, and one that I might be able to put together with him -- might be a good Scout project for him and I to work on. In the meantime, I think the flying helicopter will be a great example, and he'll have a fun time with it.   Thanks again, everyone, for the feedback in information!   DS ... View more

Hi @markbailey, @P_Bartell, @statman,     I'm happy to report that we have conducted the experiment, and it was a lot of fun! I printed out the runs and my wife had a great time conducting the experiment for us -- she managed the candies, water, coffee grounds, and recording our responses. We both stayed blindfolded the whole time, although could hear each other's responses. I'm attaching the data table, but also just commenting on some of the initial results. As a reminder, there were three factors: the actual candy color (blue, green, pink, purple), tester (A and B), and cleaning method (water swish or smelling coffee grounds)   Overall: the DOE selected each color and cleaning method equally, however shifted one run to one tester over the other, so the testers were not equally chosen. As a collective, the two testers were correct only 28% of the time, not much different than the 25% random chance (no surprise there). Pink was the least frequently predicted color. In fact in the overall heatmap, it's seen that pink was also actually NEVER correctly chosen. If there is a flavor difference, it was one that both testers mixed up with other colors. Contingency analysis results in a likelihood ratio of only 8.4, so pretty small (again, not surprising). Both testers had an equal number of correct answers (4), although unequal number of runs. Overall heatmap shows that blue was correctly chosen the most of all the colors (4 times) with green and purple being equally chosen correctly (2 times each).   By Tester: Supposedly, tester A was correct 31% of the time compared to tester B with 27%, but I think this is misrepresents tester A's ability to correctly identify the color because tester A had only 13 candies total and not 14 like tester B. Both testers had nearly identical distributions in their frequencies of color choices. So, perhaps this is an indication there could be some kind of flavor difference, but not one that either tester is trained to detect. Tester B had better results refreshing their senses using coffee grounds compared to water 75%/25%, whereas tester A was 50/50 between the two. A heatmap by tester suggests that tester B could differentiate between more colors (maybe), as that tester was able to correctly determine 3 of 4 colors, although accuracy was still low. Tester A could only correctly distinguish two colors, but also with low accuracy. A contingency by tester analysis again shows very small likelihood ratios, but indicates that tester A is twice the ratio. I'm not sure about this because of the uneven number of runs between the testers, but maybe the analysis takes this into account?   Any other thoughts would be much appreciated. By the way, I used the JMP 16EA1 to analyze the results (I'm running calculations with JMP Pro 15.0 and don't want to interrupt them).     I think the best thing that came from this is that my son is now excited about doing a DOE -- like a taste test for preference of ice cream toppings or other candies with his friends. So hey, even though we didn't really find all that much from this DOE, it got him interested in doing more and trying something new. For a 10 year old, I think it's great!   Thank you all for your input!, DS        ... View more

Hi @P_Bartell,     Thanks for the feedback and link to BIBDs. I had read a little about those before as well. However, when trying out different designs within that platform, I couldn't quite get the type of design I was thinking of like I could with the Custom DOE. Perhaps it's because I'm just not very familiar with the BIBDs. We might try that at a later date and under a different design.     I ended up going with a Custom DOE that has four levels for the colors, two participants (me and my son), and then two "cleaning" methods in-between samples (taking a drink of water or smelling coffee grounds). I think it'll be fun to do with him, and my wife is going to be the moderator and taking down the responses.     I'll be sure to update the discussion with how things went. I really appreciate everyone's feedback and thoughts on this.   Thanks!, DS ... View more

Hi @statman,     Yes, sorry about the typo. His claim was that by tasting the candy, he could tell the color of the candy shell.     Right now, we have only a single bag of the candy. I do like the idea of adding a second tester to the mix and maybe even cleansing method.     I think this will be fun, especially for him!   Thanks for the feedback.   DS ... View more

Hi @markbailey,     You're absolutely right about evidence not changing his mind! I thought it might be a fun exercise for him to get some experience with DOE and then understanding the outcomes of the analysis. And, he is more than fine with eating more!     I also thought about the change in the response column, and deleting the response limit to analyze in the contingency platform. I might actually do both so as to see the general distribution as well as the contingency.     Regarding the design, do you think I should block it for him to "take a break" so as not to influence the decision making process by tasting too much of the same chocolate? Maybe block it in groups of 5 or so -- four blocks?     Thanks for your thoughts!   DS ... View more