There is a Day of Year() function and that does not require the data to be formatted in any specific way. Underlying all date/time formats is an integer of the number of seconds elapsed since midnight, January 1, 1904. All date/time functions in JMP work based on that underlying value. However, it is curious that applying a different date format to the column has no effect. Can you verify that toggling between m/d/y and d/m/y really does not change the table? If that is the case, try restarting JMP. If that doesn't help, you need to call JMP Tech Support.
-- Cameron Willden