Fit Y by X is intended for simple but common cases involving a single dependent and single independent variable. The continuous Y role and categorical X role in your example would actually launch the Oneway platform. This platform does not work with two nested categorial variables in the X role.
So JMP provides Graph Builder for general visualization and Fit Least Squares for general modeling and testing for cases with more than one independent variable.
Fit Y by X is a specialized platform that only handles one Y and one X per analysis. The Fit Model is the platform you might need to be using, if you are looking for a mullti factor ANOVA or you may be able to get the graph you want from the Variability Chart platform
Analyze==>Quality and Process==>Variability Chart
If you can be more specific in what you are looking for out of the Fit Y by X, a better solution may be determined.