Recently convert from JMP Pro 10 to JMP Pro 11.1.1 (includes Feb 2014 maintenance update to JMP Pro 11). Two seemingly major issues:
1. Simple formula #1: Column 3 = Column 2 - Column 1. Simple formula #2: Column 5 = Column 3 - Column 4. In all cases, Column 5 should equal 0, but it does not. Any thoughts?
2. Presumably related, but: Select matching cells (Row 1 Column 3) only selects (Row 2 Column 3), not (Row 3 Column 3).
You're running into issues of numeric precision just as discussed in .
Computers can't represent floating point numbers exactly and you're seeing this reflected in your calculations and comparisons.
It can help to Round() results, especially when comparing values.
Try this in the formula for Column 5: Round( :Column 3, 1 ) - Round( :Column 4, 1 )
Round( :Column 3 - :Column 4, 0 )
Thank you for this information. I am aware this is a numeric precision issue. To quote from response #13 of that thread, by its author: "In JMP, apparently "fixed precision" is something that only exists on the screen but doesn't matter in a script. That seems fairly unhelpful to me and a rather severe violation of the principle of least surprise."
There are also multiple comments/questions in that post regarding documentation (or the absence of it) by JMP. As the post is only a few weeks old, I presume JMP 11 is being used. My example is merely illustrative and can easily by fixed with the Round feature. However, I have never encountered this issue with JMP 9, JMP 10, or Excel...
Does anyone know if this issue is more problematic in JMP 11 for some reason? Does JMP have documentation on this? Or, more importantly, is there a simple way to make fixed precision some sort of default? This really is a major issue if the Round feature must now be incorporated into even the most basic functions to make them work correctly.
It is an Intel issue not microsoft or SAS or any other program. There is even an IEE-standard.
floating point double-precision is what Intel is using as standard at the moment. Precision is 15 digits.
Welcome in the digital age and the caveats.
"There are 10 types of people in the world; those that understand BINARY and those that do not"..
What there could be the case is some kind of FUZZ factor. That is rounding errors or very small digits are faked in a way you do not see the difference.
Excel and every other tool is having this same kind of behavior. That you do not see it, as the first calculators did, is smart faking.
Amazing how many people have forgotten what are the basics behind all this. I remember a slider (Lorenz chaotic ssytems), now I am expecting a reaction hhhuu what is that?
Thank you both for the additional replies. I take it from all three of these replies that there is no ubiquitous, Round preset feature to select in JMP 11 ("smart faking"), so I suppose this concludes this thread.