The formula in the data table doesn't know the selection of the Local Data Filter of your report.
More surprisingly, this is also true for a transform column in a report - it just ignores the selection of the local data filter of the respective report.
We have become so accustomed to what the summary statistics in graph builder do - automatically.
With this background, the behavior of Column Formulas feels like a bug:
Transform Column: Bug with excluded rows
With the words of Jmp Support (TS-00056243)
It appears the transform column formula is not re-evaluated in response to Local Data Filter changes like it is for Global Data Filter changes. I will report this as a bug. Thank you for pointing it out.
After a while I got the feedback:
The issue has been reported and as you assume, development shares that this is behaving as designed, where the transform columns behave just like regular table columns with respect to honoring row states.
Concerning future versions of Jmp, there is an entry in the wish list which suggests to add an option to not ignore the local data filter:
Transform Columns - as comfortable as Summary Statistics?
As a workaround with current Jmp, you have to use a REAL column (not a transform column (*)), ask the Local Data Filter for the selected rows via JSL and use the response as an input for the formula:
https://community.jmp.com/t5/Discussions/Transform-Column-Bug-with-excluded-rows/m-p/676095/highligh...