Okay, a better solution:
1. Make the ellipse plot with the points graph element as well
2. Select one of the data points from each source and then right-click > Rows > Label
You should have a label for each source ellipse now - assuming you have defined the source colunm as a label)
Now you want to get rid of the points from the plot. Actually we will just make them invisible.
3. Select the rows for the points that define the ellipses and define the marker for the rows as a space character, " " (Rows > Markers > Others...)
See the example data table with Graph Builder script.