1. When I make a regression, and find that the regression model is poor, for instance, residual vs predicted Y is a speaker-shape, in JMP, what method can help me to determine the data transformation
2. in Minitab, I can use macro % Boxcoxregression, and find an appropriate Lambda value for data transformation, I'd like to if JMP also has a similar method
Under the red triangle you will find the capability to transform the model.
Adding to Lou's response, for more details, see: http://www.jmp.com/support/help/Factor_Profiling.shtml.
You might also care to look at: https://community.jmp.com/docs/DOC-6875.