There must be something else going on, as your syntax for Col Mean is OK. For example, the following script works:
Names Default To Here(1);
dt = as table(J(50, 1, randomuniform()), << column names ({"RLU"}) );
dt << new column ("Assay", character, << set values ( words("abcd", "")[J(50, 1, randominteger(1,4))]));
dt << new column ("mean", formula(colmean(:RLU, :Assay)));
You can see that it is computing group means.
When I first ran my code, I forgot to place the column name RLU in curly braces, so the column name was not created properly, and I got an error like the one you described. Perhaps it has something to do with this?