I have added its OSI documentation to JMP17. You should be able to find it under OSI > Objects > Fit Model > Model Dialog > Item Messages > Effects.
For FactorialToDegree and also PolynomialToDegree, what you did is not what you think it is. The arguments in side of the Macro names are interpreted as columns. So in your example, the number 2 is interpreted as the second column, which is :age. And FactorialToDegree and PolynomialToDegree will refer to the value in the Degree number edit box as the value for degree.
In JMP 16 and previous versions, you may have to go through display box scripting to change that. E.g.
Names default to here( 1 );
dt = open("$SAMPLE_DATA\Big Class.jmp");
dialog = dt << Fit Model(
Personality( "Standard Least Squares" ),
Emphasis( "Effect Leverage" ),
);
report(dialog)[NumberEditBox(1)] << set (3);
dialog << Effects(FactorialtoDegree(:height, :weight, :age));
I made small improvement in JMP17. You will see it in the Effects OSI item when you have JMP17.