Try resizing the graph (make it larger) before saving the image rather than changing the dpi of the saved image. With a script you can make the image much larger than your monitor size.
Names default to here(1);
dt = Open("$SAMPLE_DATA/iris.jmp");
s3d = dt << Scatterplot 3D(
Y( :Sepal length, :Sepal width, :Petal length, :Petal width ),
Frame3D( Set Graph Size( 2000, 2000 ) )
);
s3d << Save Picture("$DESKTOP/image.png","png");