Have you tried using the "?" tool and clicking on the Results Table you are interested in finding the details on?
To do this, either select the "? Help" tool from the Tools pull down menu and then click on the table you are interested in, or input Shift/? to bring the tool up, and then click on the table of interest.
It is a good start on finding out the details.
Here is a section of the JMP help that might be of use:
Without seeing your particular results I can't be sure what is happening, but I suspect the model you're fitting has a number of random effects that are likely close to zero/absorbing no variance. When fitting a mixed model, you have the option to allow "unbounded variance components," which means that a variance component can converge to a negative estimate. This might seem strange (or downright ridiculous) since we know in reality a variance can never be negative. But, it's been shown that constraining estimates for the variance components to be positive leads to bias in estimation of the fixed effects (if you're interested in the statistical reasons for this I've included a google scholar search below).
So, what does that have to do with your coefficient of determination for the full model? If many of your variance components are estimated to be negative (meaning they're likely inactive in the population), and whatever fixed effects you're estimating absorb little variance, it's possible for the overall model R2 to be negative.
As I said before, it's hard to know what's happening exactly without seeing more about your particular situation, but hopefully this helps put you on the right path!
Your description is correct in that I have a random effect with a negative estimate, here is a screen shot (should have included this in the original post).
I was wondering about the statistical details for R2 for mixed models (which I can't find in the link you provided). Based on a quick internet search it looks like there is more than one method to calculate R2 for mixed models.
Sorry, I clearly didn't answer the question you asked :) I just looked through the documentation again and although there are statistical details for R^2 for fixed-effects only models, I also can't seem to find details for models with random effects. I will do some more digging, but I would suggest you email firstname.lastname@example.org with your question and they will likely be able to get the answer to your question quickly.
Is it reporting the Generalized RSquare as described in the Mixed Model Personality (JMP Pro only, which I don't have)? This excerpt is from the JMP Fitting Linear Models Book under the Help menu. It looks like this would be reasonable although I don't get a negative R2 when I try this calculation: