Yes! I believe, as usual, that you are right regarding the weird nature of trying to predict a categorical variable, in which you effectively need two equations, thereby making things trickier. What I hope to do is eventually have a large enough sample size to where I have a true gradient, either 1-10, 1-5, or 1-100, the idea being the same: 1=terrible, the highest number=great. Even in this case, I either have two categories (bleaching-resistant vs. bleaching-sensitive) or sometimes three (the former two + "intermediate"). I might play around with recoding them as 1, 50, and 100 for susceptible, intermediate, and resistant, respectively, and see what kinds of results I get. I think the more samples I get and analyze, the more sense it makes to have my Y be a continuous variable. With 20 samples I'm playing with now, I don't have enough data to go beyond 2-3 bins essentially. But this will be interesting to see if my models look very different when I recode my categorical classifications into a continuous variable. It would surely make all the model-building more seamless. And in reality, in nature it will NOT be as simple as "healthy" vs. "sick;" I DO expect to pick up a gradient that spans a health spectrum like you'd see in humans (for instance, what you see in the JMP diabetes dataset). The funny thing about this exercise, though, regarding the probability score, is that I never really looked at these much; I went directly to JMP's "guess." But now looking at the actual probabilities is super interesting because that is ultimately what I want to report: coral A has a 95% chance of bleaching. So through this model averaging exercise, I accidentally picked up some cool nuances that I totally missed before! And like I said before, even if I am "stuck" with categorial responses for the time being, the model averaging feature of Formula Depot works fine for this.
You'll be happy to know that, thanks to your GUI, there are over 20,000 neural network models (21,450, but who's counting?!) in my recently submitted paper (which describes essentially phase 1 = the lab experiment). Obviously, only a handful passed all quality control, with even fewer discussed at length in the text, but it was extremely useful to see which NN parameters are essentially most important for me to tune. Also, I want to build the simplest, (potentially) most parsimonious model, so if one NN has two hidden layers, each with three activation nodes, whereas a second model has one layer with NTANH(3), all else being equal, I'd choose the latter, though I guess this isn't really critical since even the simplest one is pretty complex. One thing I sought to learn this year at Discovery, which I did thanks to Chris Gotwalt, is whether the variable importance metrics under the NN Profiler can be used to essentially tell you which analytes were most influential in the model. The old rule was that NNs are a black box, and it's way too complicated to mine which response variables were most important, but I'm glad to see this is not entirely true. Are they hard to interpret? Yes. Can you say one protein's behavior scales in a particular, easily defined way with the model's predictive power? Maybe not. But if a molecular biologist asked me: "Which coral proteins were most important in your best NN model" I could at least list off a few from the variable importance feature of the NN profiler. This is actually why I chose to discuss the gen-reg model in my talk; I wasn't sure if I could "trust" the profiler for a NN (!), but now I see that I can. But anyway, this is why, as a modeling novice, I benefit so much from these sorts of symposia. Thanks again for your well thought-out response and all your help in general.
Anderson B. Mayfield
