Maybe try this without Graph Builder.
Use Fit Y by X. Put your X and Y values in the proper roles.
In the resulting graph, click the red triangle and choose Fit Line.
Now go back to the red popup and choose Group By... and specify your Group column.
One more trip to the red popup and choose Density Ellipse > 0.95.
To make things prettier you will likely want to right-click the graph and add a Row Legend with the Group column to establish the marker colors.
You will also want to click red popups associated with each density ellipse to shade them, if you like the shading.
I've included my result for your sample data.
Dan Obermiller