I completely understand -- it's wonderful that Graph Builder automatically updates the display when dragging columns over a particular role; but, sometimes that's problematic, as in your case when you have many columns you're trying to drop in a different role. In situations like these, what I like to do is select the columns in the column list, but rather than drag the columns to a particular role, I simply click the role name. As far as Graph Builder is concerned that is the same as dragging the columns to that role.
Hope this helps!
Julian