I have not found a specific way to adjust the font size of the group labels. Label size seems to be controlled by the the text font preferences, i.e. they can be decreased by choosing a smaller size in preferences/fonts/text. But that will affect the label size in other reports as well as some other elements, legend text size for example. Although more general than desired, this preference has more restricted effects than the global shortcut (cmd+- or Make Text Smaller in the View Menu). For a particular graph it still can be useful.
Preferences can also be changed with jsl. Here's an example how to decreases size while retaining font type.
// Run first two lines once
p_expr = Expr( Get Preferences( text font ) );
p_start = p_expr;
// Repeat following two lines until font size of group labels are OK
p = p_expr;
Preferences( text font( Arg( Arg( p, 1 ), 1 ), Arg( Arg( p, 1 ), 2 ) - 1 ) );
// Run this to reset
Eval( p_start );