Multiple Comparisons (2 of 3)

---

Since Scheffé's test can also be used to test multiple comparisons, it is important to use the Scheffé's test if it is more powerful than the Bonferroni adjustment. Scheffé's test will be more powerful if more than three comparisons are planned among three means or more than seven comparisons are planned among four means. In almost all realistic situations, the Bonferroni adjustment is more powerful.

As in the case of pairwise comparisons, there is a tradeoff between controlling the EER and power. The Bonferroni adjustment provides control over the EER at a substantial cost in power. Some statisticians argue that it is not always necessary to control the EER. The "experiment," after all, is an arbitrary unit of analysis. Why is it necessary to control the error rate in one experiment but not in a whole series of experiments. For example, if a researcher conducted a series of five related experiments, few statisticians would recommend that the probability of a Type I error in any of the comparisons in any of the experiments be controlled. Nonetheless, the experiment is a convenient unit of analysis. The general consensus is that steps should be taken to control the EER.