First of all, I am very happy if anybody suggests any different / better way of achieving a goal, so thanks for the question, statman.
The situation is that there are10 test machines that perform a final control on the products. The machines measure several parameters. There are many different product types that are all tested in the same way. The product types have different materials / constructions so differences in the parameter statistics between products types are to be expected. However, the parameter statistics should not differ across the machines (meaning that if I could ensure that each machine tested the same amount of each product, I would expect to see the same mean / standard deviation on each machine).
But I do not see the same mean of standard deviation across the machines, firstly because the amount of each product tested by each machine is not controlled. So the question is "is the difference in the statistics between machines for each parameter solely due to differences in the numbers of different products being tested?" I want to be able to answer this question in a live production environment at any time.
Here's my XBar S Chart
To answer you question, in the subgroup, I have all data from the specific machine for a specific product over tested over a specific time-period. In the case above, I would say Machine 17 looks "questionable" on this parameter as it is reading low. I look for many signals, not just one, so I would now check Machine 17 on several products (first similar products, then products with a different construction), if I was seeing this pattern in several cases, Machine 17 needs examining further.
Another way I thought of doing this was to perform p-Tests automatically, so I would first get my population statistics for a parameter then look at the sample of this population collected by each machine and identify any significant differences. My problem here was I couldn't work out how to calculate this automatically.
Any comments or criticisms of how I am using the statistics are welcome. I am a willing amateur when it comes to stats.