dt:sex << set property("value ordering", {"M", "F"} );
this work as alternative to explicitly spell the Custom Order.
I am not sure why omitting the "ing" such that dt:sex << set property("value order", {"M", "F"} ); failed, despite that is what stated in JMP's documentation.