The column functions operate on the whole table, ignoring selection and exclusions. However, you can make a formula column that's dependent on the row state and use that column as a BY variable.
New Column( "mysel", Formula( Selected( Row State() ) ) );
New Column( "meany", Formula( Col Mean( :y, :mysel) ) );
If you're using the global data filter with selection turned on, the Column Means will be recomputed whenever the selection changes.